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ABSTRACT 



The papers presented at this symposium explore each 
component of the National Assessment of Educational Progress (NAEP) , from the 
development of the assessment frameworks through the reporting of assessment 
results. The papers are: (l) "NAEP Frameworks and Achievement Levels" (Robert 

A Forsyth) ; (2) "Assembly of Test Forms for Use in Large-Scale Educational 

Assessments" (Wim J. van der Linden) ; (3) "A Brief Introduction to Item 

Response Theory for Items Scored in More Than Two Categories" (David Thissen, 
Kathleen Billeaud, Lori McLeod, and Lauren Nelson) ; (4) "Some Ideas about 

Item Response Theory Applied to Combinations of Multiple-Choice and 
Open-Ended Items: Scale Scores for Patterns of Summed Scores" (Kathleen 
Billeaud, Kimberly Swygert, Lauren Nelson, and David Thissen) ; (5) "Enhancing 

the Validity of NAEP Assessment Level Score Reporting" (Ronald K. Hambleton) ; 
(6) "1998 Civics and Writing Level-Setting Methodologies" (ACT, Inc.); (7) 

"The Criticality of Consequences in Standard Setting: Six Lessons Learned the 
Hard Way by a Standard-Setting Abettor" (W. James Popham) ; and (8) 
"Acknowledgments and Appendices." Two appendixes list participants and 
present the conference agenda. (SLD) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



What S§ The Nation's 

Report Card? 

The Nation's Report Card, the National Assessment of Educational Progress (NAEP), is the only nationally representative and 
continuing assessment of what America's students know and can do in various subjects. Since 1969, assessments have been 
conducted periodically in reading, mathematics, science, writing, history/geography, and other fields. By making objective 
information on student performance available to policymakers at the national, state, and local levels, NAEP is an integral part 
of our nation's evaluation of the condition and progress of education. Only information related to academic achievement is 
collected under this program. NAEP guarantees the privacy of individual students and their families. 

NAEP is a congressionally mandated project of the National Center for Education Statistics, the U.S. Department of Education. 
The Commissioner of Education Statistics is responsible, by law, for carrying out the NAEP project through competitive 
awards to qualified organizations. NAEP reports directly to the Commissioner, who is also responsible for providing 
continuing reviews, including validation studies and solicitation of public comment, on NAEP's conduct and usefulness. 

The National Assessment Governing Board (NAGB) was established under section 412 of the National Education Statistics 
Act of 1994 (Title IV of the Improving America's Schools Act of 1994, P.L. 103-382). The Board was established to formulate 
policy guidelines for NAEP. The Board is responsible for selecting subject areas to be assessed, developing assessment 
objectives, identifying appropriate achievement goals for each grade level and subject tested, and establishing standards 
and procedures for interstate and national comparisons. 
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The National Assessment Governing Board ; s policy on student performance standards 
states that the achievement levels should influence all aspects of the National 
Assessment of Educational Progress (NAEP) assessments, from the development of 
the assessment frameworks through the reporting of assessment results. The purpose 
of these proceedings is to explore each component of the assessment, from drawing 
board to Boardroom, in order to understand more fully the relationships between the 
performance standards and these assessment components, and the mutual impacts 
they have on each other. 

In Section 1 Robert Forsyth from the University of Iowa examines the relationship 
between the assessment frameworks and the levels. He discusses the general purposes 
of the frameworks and the preliminary achievement level descriptions, those state- 
ments of content that students should know and be able to do at each level. His 
paper also examines selected characteristics of the frameworks, e.g., item formats, 
breadth of coverage, and the complexity of the cognitive dimensions, and how such 
characteristics influence both the preliminary and final achievement level descriptions. 

Wim van der Linden from the University of Twente in the Netherlands describes two 
test assembly procedures used in large-scale assessments in Section 2. One procedure 
assigns items to forms in units called blocks as in NAEP, while the second method 
assigns unique items from the item pool to test forms. The author discusses the rela- 
tionship between the characteristics of the assessment, such as size of the item pool 
and the number of desired forms, and the recommended methodology for test assembly. 

In Sections 3 and 4 David Thissen and his colleagues at the University of North 
Carolina at Chapel Hill provide an introduction to item response theory for scoring 
assessments such as NAEP which have both multiple choice items as well as multiple- 
scored items. The relationship between the score scales and the standard setting 
methodologies is important since setting performance standards is impacted by data 
coming from different item types. Most NAEP assessments use mixed item formats, 
except the writing assessment, which employs only extended constructed responses 
to prompts. How these assessments are scored and scaled can have significant conse- 
quences for the standard-setting results. 

In Section 5 Ronald Hambleton from the University of Massachusetts at Amherst 
explores the issue of score reporting in NAEP, and presents the results of a small-scale 
study of the understandability of NAEP score reports. Hambleton also provides some 
guidance on how to improve NAEP score reports and comments on the usefulness of 
market-basket reporting for NAEP, an index similar to the Consumer Price Index. 

Section 6 provides the reader with a look at the proposed methodologies for devel- 
oping the student performance standards on the 1998 NAEP civics and writing 
assessments. This section outlines the key features of the 1998 proposal and the 
overarching principles for developing the levels. 
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V 



Finally, in Section 7, W. James Popham, professor emeritus from the University of 
California at Los Angeles, provides a commentary on the criticality of consequences 
in standard-setting. In his inimitable style, Popham offers six lessons learned the hard 
way by a standard-setting abettor. Popham calls on his many years of experience in 
implementing standard setting initiatives for more than three dozen high-stakes tests 
for students, teachers, and administrators, and shares his wisdom on the subject. 

The National Assessment Governing Board hopes that the issues raised by the 
authors, and the solutions proposed, will extend the national conversation on stan- 
dard setting, and will benefit not only the National Assessment, but all those individ- 
uals and agencies whose responsibilities includes setting performance standards in 
various academic subjects. 
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